Tracking The Sensitivity of The Learning Models Toward Exact and Near Duplicates

Most real-world datasets contaminated by quality issues have a severe effect on the analysis results. Duplication is one of main that hinder these Different studies tackled duplication issue from different perspectives. However, revealing sensitivity supervised and unsupervised learning models under existence types duplicates, deterministic probabilistic, not broadly addressed. Furthermore, simple metric used to estimate ratio both duplicates regardless probability which record considered duplicate. In this paper, five classifiers four clustering algorithms toward probabilistic with ratios (0% - 15%) tracked. Five evaluation metrics are accurately track changes in each model, MCC, F1-Score, Accuracy, Average Silhouette Coefficient, DUNN Index. Also, measure within dataset introduced. The results revealed effectiveness proposed reflects dataset. All models, classification, differently sensitive duplicates. RF Kmeans positively affected means their performce increase as percentage increases. rest existence, especially high negatively affect performance.